Generate: remove flakyness in test_generate_from_inputs_embeds_decoder_only#33602
Merged
ydshieh merged 1 commit intohuggingface:mainfrom Sep 20, 2024
Merged
Generate: remove flakyness in test_generate_from_inputs_embeds_decoder_only#33602ydshieh merged 1 commit intohuggingface:mainfrom
test_generate_from_inputs_embeds_decoder_only#33602ydshieh merged 1 commit intohuggingface:mainfrom
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Contributor
Author
|
hah, a PR to fight flakiness got stuck in another flaky test 🙃 |
ydshieh
reviewed
Sep 20, 2024
| self.assertListEqual(outputs_from_ids.sequences.tolist(), outputs_from_embeds.sequences.tolist()) | ||
|
|
||
| # But if we pass different inputs_embeds, we should get different outputs | ||
| torch.manual_seed(0) |
Collaborator
There was a problem hiding this comment.
very nice to remove torch.manual_seed!
ydshieh
approved these changes
Sep 20, 2024
Collaborator
ydshieh
left a comment
ydshieh
left a comment
There was a problem hiding this comment.
Great! I am up to merge as it is.
Another flaky test is worth another PR, and let's have fewer failures progressively.
But up to you :-)
Contributor
Author
|
@ydshieh I don't have merge permissions, and I agree with you -- can you merge this PR for me? :) |
BernardZach
pushed a commit
to BernardZach/transformers
that referenced
this pull request
Dec 5, 2024
…er_only` (huggingface#33602) almost zero is not zero
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
test_generate_from_inputs_embeds_decoder_onlyhad low probability of failure... but not zero (see #33593 and its CI runs)Instead of assuming that calling
generatewith randominputs_embedswill result in a different text, let's check the scores instead. Even if the output text is the same (unlikely), the token scores are almost guaranteed to be different.